Contextual Gaussian Process Bandit Optimization

How should we design experiments to maximize performance of a complexsystem, taking into account uncontrollable environmental conditions? Howshould we select relevant documents (ads) to display, given information about theuser? These tasks can be formalized as contextual bandit problems, where at eachround, we receive context (about the experimental conditions, the query), andhave to choose an action (parameters, documents). The key challenge is to tradeoff exploration by gathering data for estimating the mean payoff function over thecontext-action space, and to exploit by choosing an action deemed optimal basedon the gathered data. We model the payoff function as a sample from a Gaussianprocess defined over the joint context-action space, and develop CGP-UCB, anintuitive upper-confidence style algorithm. We show that by mixing and matchingkernels for contexts and actions, CGP-UCB can handle a variety of practical ap-plications. We further provide generic tools for deriving regret bounds when usingsuch composite kernel functions. Lastly, we evaluate our algorithm on two casestudies, in the context of automated vaccine design and sensor management. Weshow that context-sensitive optimization outperforms no or naive use of context.